Assignment #1

library(tidyverse)         # for graphing and data cleaning
library(tidymodels)        # for modeling
library(naniar)            # for analyzing missing values
library(vip)               # for variable importance plots
library(glmnet)            # for regularized regression, including LASSO

hotels <- readr::read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-02-11/hotels.csv')

Setting Up Git and Github in RStudio

Here is my Github link.

Creating a Website

• Website Link
  • Here is the link to my website.
• Building a Career in Data Science, Chapter 4: Building a Portfolio reflection
  • One thing that stood out most to me was the comment about making a portfolio with results that regular people can digest. I believe that this is important for multiple reasons. First, making sure that your portfolio is understandable will help recruiters who are not necessarily versed in data science techniques understand your work and your abilities. Second, it is most likely that when working in the professional world, you will need to explain your results to people who are not on the data science team, so it is good to practice this ability.

Machine Learning review and intro to tidymodels

1. Read about the hotel booking data, hotels, on the Tidy Tuesday page it came from. There is also a link to an article from the original authors. The outcome we will be predicting is called is_canceled.

• Without doing any analysis, what are some variables you think might be predictive and why?
  • There are a few variables that could be predictive, however previous_cancellations definitely stands out. It is reasonable to assume that if someone has canceled before, they could possiblky cancel again. booking_changes could also be predictive, as someone who makes a bunch of changes is likely to be unsure about their booking and one of the changes that they could make could be canceling the stay. Finally, a third variable that could be predictive is customer_type, as the type of customer could make it easier or more difficult to cancel the stay.
• What are some problems that might exist with the data? You might think about how it was collected and who did the collecting.
  • One issue with the way that the data was collected is that there are almost twice as many observations for the city hotel as the resort hotel, which could introduce bias into the data. The data for canceled bookings could also be less accurate, as variables such as adults, children, and babies could be inaccurate due to the family or group never showing up. Therefore, there could be bias towards non-canceled bookings based on how the data was collected. The reservation_status variable is also a bit redundant, as it states whether someone canceled their reservation or not, which is already given in the is_canceled variable. Finally, each reservation is missing a unique identifier, so
• If we construct a model, what type of conclusions will be able to draw from it?
  • If we construct a model, the type of conclusions that we’ll be able to draw from it are likely to be which variables are most important to determine the likelihood of cancellation. This could easily be achieved using the LASSO technique to analyze variable importance.

2. Create some exploratory plots or table summaries of the data, concentrating most on relationships with the response variable. Keep in mind the response variable is numeric, 0 or 1. You may want to make it categorical (you also may not). Be sure to also examine missing values or other interesting values.

hotels %>% 
  ggplot(aes(x = hotel)) + 
  geom_bar(fill = "blue") + 
  facet_wrap(vars(is_canceled))

hotels %>% 
  ggplot(aes(x = customer_type)) + 
  geom_bar(fill = "red") +
  facet_wrap(vars(is_canceled))

hotels %>% 
  ggplot(aes(x = previous_cancellations)) +
  geom_bar(fill = "orange") +
  xlim(0,3) +
  ylim(0,7000) +
  facet_wrap(vars(is_canceled))

hotels %>% 
  ggplot(aes(x = adults)) + 
  geom_bar(fill = "green") +
  xlim(0,5) +
  facet_wrap(vars(is_canceled))

hotels %>% 
  ggplot(aes(x = children)) + 
  geom_bar(fill = "green") +
  xlim(0,5) +
  ylim(0,2500) +
  facet_wrap(vars(is_canceled))

hotels %>% 
  ggplot(aes(x = babies)) + 
  geom_bar(fill = "green") +
  xlim(0,5) +
  ylim(0,200) +
  facet_wrap(vars(is_canceled))

3. First, we will do a couple things to get the data ready, including making the outcome a factor (needs to be that way for logistic regression), removing the year variable and some reservation status variables, and removing missing values (not NULLs but true missing values). Split the data into a training and test set, stratifying on the outcome variable, is_canceled. Since we have a lot of data, we’re going to split the data 50/50 between training and test. I have already set.seed() for you. Be sure to use hotels_mod in the splitting.

hotels_mod <- hotels %>% 
  mutate(is_canceled = as.factor(is_canceled)) %>% 
  mutate(across(where(is.character), as.factor)) %>% 
  select(-arrival_date_year,
         -reservation_status,
         -reservation_status_date) %>% 
  add_n_miss() %>% 
  filter(n_miss_all == 0) %>% 
  select(-n_miss_all)

set.seed(494)

hotel_split <- initial_split(hotels_mod,
                             prop = 0.5)
hotel_train <- training(hotel_split)
hotel_test <- testing(hotel_split)

4. Pre-processing

hotel_recipe <- recipe(is_canceled ~.,
                       data = hotel_train) %>% 
  step_mutate_at(children, babies, previous_cancellations,
                 fn = ~ifelse(. > 0, 1, 0)) %>% 
    step_mutate_at(agent, company,
                  fn = ~ifelse(. == "NULL", 1, 0)) %>% 
      step_mutate(country = fct_lump_n(country, 5)) %>% 
        step_normalize(all_numeric()) %>% 
          step_dummy(all_nominal(), -all_outcomes())

hotel_recipe %>% 
  prep(hotel_train) %>% 
  juice()

5. LASSO model and workflow

• We would want to use a LASSO workflow because LASSO uses an importance coefficient which reduces to zero when the variable is deemed to be not predictive of our outcome variable. This will allow us to reduce the size of our dataset and only focus on our indicator variables that matter.

lasso_hotel_mod <- logistic_reg(penalty = tune()) %>%
  set_engine("glmnet") %>% 
    set_mode("classification")

hotel_workflow <- workflow() %>%
  add_recipe(hotel_recipe) %>%
    add_model(lasso_hotel_mod)

lasso_hotel_fit <- hotel_workflow %>%
  fit(data = hotel_train)

lasso_hotel_fit

## ══ Workflow [trained] ══════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: logistic_reg()
## 
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 5 Recipe Steps
## 
## ● step_mutate_at()
## ● step_mutate_at()
## ● step_mutate()
## ● step_normalize()
## ● step_dummy()
## 
## ── Model ───────────────────────────────────────────────────────────────────────
## 
## Call:  glmnet::glmnet(x = maybe_matrix(x), y = y, family = "binomial") 
## 
##    Df  %Dev   Lambda
## 1   0  0.00 0.232900
## 2   1  2.92 0.212200
## 3   1  5.29 0.193300
## 4   1  7.23 0.176200
## 5   1  8.86 0.160500
## 6   1 10.24 0.146300
## 7   1 11.42 0.133300
## 8   1 12.43 0.121400
## 9   2 13.34 0.110600
## 10  2 14.38 0.100800
## 11  2 15.29 0.091850
## 12  3 16.24 0.083690
## 13  5 17.38 0.076260
## 14  5 18.58 0.069480
## 15  7 19.68 0.063310
## 16  7 21.76 0.057690
## 17  7 23.59 0.052560
## 18  7 25.20 0.047890
## 19  8 26.66 0.043640
## 20  9 27.97 0.039760
## 21 10 29.26 0.036230
## 22 10 30.40 0.033010
## 23 11 31.43 0.030080
## 24 12 32.37 0.027410
## 25 13 33.21 0.024970
## 26 14 33.99 0.022750
## 27 14 34.68 0.020730
## 28 15 35.29 0.018890
## 29 18 35.86 0.017210
## 30 19 36.41 0.015680
## 31 21 36.90 0.014290
## 32 22 37.34 0.013020
## 33 23 37.74 0.011860
## 34 23 38.09 0.010810
## 35 25 38.40 0.009849
## 36 29 38.74 0.008974
## 37 31 39.07 0.008177
## 38 32 39.37 0.007451
## 39 34 39.66 0.006789
## 40 36 39.93 0.006186
## 41 40 40.18 0.005636
## 42 41 40.42 0.005135
## 43 42 40.63 0.004679
## 44 43 40.82 0.004263
## 45 44 41.00 0.003885
## 46 44 41.17 0.003540
## 
## ...
## and 40 more lines.

6. Tune and fit the model

set.seed(494)

hotel_cv <- vfold_cv(hotel_train, v = 5)

hotel_lasso_pen_grid <- grid_regular(penalty(), levels = 10)

hotel_lasso_tune <- hotel_workflow %>% 
  tune_grid(resamples = hotel_cv,
            grid = hotel_lasso_pen_grid)

hotel_lasso_tune %>% 
  collect_metrics() %>% 
      filter(.metric == "accuracy")

hotel_lasso_tune %>% 
  collect_metrics() %>% 
      filter(.metric == "accuracy") %>% 
        ggplot(aes(x = penalty, y = mean)) +
        geom_point() +
        geom_line() +
        scale_x_log10(
        breaks = scales::trans_breaks("log10", function(x) 10^x),
        labels = scales::trans_format("log10",scales::math_format(10^.x))) +
        labs(x = "Penalty",
             y = "Accuracy")

best_param <- hotel_lasso_tune %>% 
  select_best(metric = "accuracy")

hotel_lasso_final_workflow <- hotel_workflow %>% 
  finalize_workflow(best_param)

hotel_lasso_final_mod <- hotel_lasso_final_workflow %>% 
  fit(data = hotel_train)

hotel_lasso_final_mod %>% 
  pull_workflow_fit() %>% 
  tidy()

• Some of the coefficients are zero, signifying that those predictor variables are not a factor in our LASSO model.

7. Variable Importance Graph

hotel_lasso_final_mod %>% 
  pull_workflow_fit() %>% 
  vip()

• It is unsurprising that the type of room reserved is the most important variable. It can explain a possible hypothesis that the majority of cancellations come when someone in a group drops out of a reservation. The importance of the non-refundable deposit type is also unsurprising because putting down a non-refundable deposit will most likely influence someone to not cancel their reservation.

hotel_lasso_test <- hotel_lasso_final_workflow %>% 
  last_fit(hotel_split)

hotel_lasso_test %>% 
  collect_metrics()

• The accuracy estimate for the test data is slightly higher than for the cross-validated data.

preds <- collect_predictions(hotel_lasso_test)

preds %>% 
  conf_mat(.pred_class, is_canceled)

##           Truth
## Prediction     0     1
##          0 34492  3165
##          1  8004 14032

• Sensitivity: \(\frac{34492}{34492+8004} = 0.81165\)
• Specificity: \(\frac{14032}{14032 + 3165} = 0.815956\)

preds %>% 
  ggplot(aes(x = .pred_1, fill = is_canceled)) +
  geom_density(alpha = 0.5, color = NA)

• For an accuracy close to 1, the plot would have almost no overlap at all, as the true positive and true negative rates would each be close to 1.
• If we want to have a higher true positive rate, we should make the cutoff higher than 0.5, as only including predictions that are more likely to be canceled would filter out predicitions that were somewhat likely (i.e. 50%-60% likely) to be canceled that ended up not being canceled.
• If the true positive rate increases, the true negative rate should increase as well, because by increasing the true positive rate, we are eliminating incorrect predictions. Eliminating incorrect predictions will drive down the false negative rate.

8. Let’s say that this model is going to be applied to bookings 14 days in advance of their arrival at each hotel, and someone who works for the hotel will make a phone call to the person who made the booking. During this phone call, they will try to assure that the person will be keeping their reservation or that they will be canceling in which case they can do that now and still have time to fill the room. How should the hotel go about deciding who to call? How could they measure whether it was worth the effort to do the calling? Can you think of another way they might use the model?

• The hotel should first call people who are in room type P, as room type P is the most important variable when it comes to predicting whether a reservation will be canceled or not. The hotel should also look at the type of deposit that the reservation maker put down, as that is also very predictive of whether the reservation will be canceled or not. The way that they could measure whether the calls were worth it or not could be by looking at how many cancellations were discovered based on the calls versus the total number of cancellations. They could also observe how many reservations were made for the rooms after they were canceled as a result of the calls. Another way that they could use the model is by looking at the rooms that are canceled the most and making deposits on those rooms non-refundable.

9. How might you go about questioning and evaluating the model in terms of fairness? Are there any questions you would like to ask of the people who collected the data?

• In terms of fairness, we might want to discuss the potential issues with who we are evaluating. With regards to the individual, it is possible that we are only looking at wealthy people who can afford to go on vacation and stay at a hotel or a resort. With regards to the group reservation type, it is possible that the majority are either multiple wealthy friends, or families who are able to afford going on vacation. This is a situation where demographics are important, and I would want to ask the researchers about the demographics of the people who were making the reservations, as by including mostly wealthy people, the researchers would not be understanding the true trends of the population.

Bias and Fairness Reflection

• Throughout watching this lecture, I was not necessarily surprised by anything, but I did find it interesting how a lot of the biases used race as an example. I did not realize how much of an issue racial bias was in data science, and how large organizations such as Facebook, Amazon, and IBM are contributing to it so much. This lecture also helped me understand why it is so important to pay attention to bias in machine learning and when writing algorithms and I came to this conclusion: identifying biases can help people figure out when their algorithm is wrong. Identifying the potential biases in an algorithm can help a researcher find what, or who, their algorithm is omitting.